Image processor, image processing method, and imaging device

ABSTRACT

An image processor includes a first correction section that calculates a luminance average value of an image and corrects a luminance of the image on a basis of a periodic change in the luminance average value, and a second correction section that acquires color information on the image and corrects the color information on a basis of a periodic change in the color information. This configuration suppresses a flicker in imaging at a high-speed frame rate.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase of International PatentApplication No. PCT/JP2018/039696 filed on Oct. 25, 2018, which claimspriority benefit of Japanese Patent Application No. JP 2017-236879 filedin the Japan Patent Office on Dec. 11, 2017. Each of theabove-referenced applications is hereby incorporated herein by referencein its entirety.

TECHNICAL FIELD

The present disclosure relates to an image processor, an imageprocessing method, and an imaging device.

BACKGROUND ART

PTL 1 listed below has disclosed, for example, that a correction valuefor flicker correction is calculated for each of color components of aninputted image, and the flicker correction is performed for each of thecolor components of the inputted image.

CITATION LIST Patent Literature

PTL 1: Japanese Unexamined Patent Application Publication No. 2016-82510

SUMMARY OF THE INVENTION Problem to be Solved by the Invention

However, in the technique described in PTL 1, the correction value iscalculated for the input image to perform the correction by means ofimage processing, and thus modification is performed using a digitalgain. Accordingly, there is an issue in which an image with a blown-outhighlight or a blocked-up shadow is not able to be corrected due to alarge flicker amplitude.

In addition, in a case where a frame rate of imaging is high withrespect to a flicker frequency, for example, a shutter period of animaging element is shorter than a flicker period. Accordingly, it is notpossible to perform a method of suppressing a flicker by setting shuttertime to an integer multiple of the flicker period.

Accordingly, it has been requested to suppress a flicker in imaging at ahigh-speed frame rate.

Means for Solving the Problem

According to the present disclosure, there is provided an imageprocessor including: a first correction section that calculates aluminance average value of an image and corrects a luminance of theimage on a basis of a periodic change in the luminance average value;and a second correction section that acquires color information on theimage and corrects the color information on a basis of a periodic changein the color information.

According to the present disclosure, there is provided an imageprocessing method including: calculating a luminance average value of animage and correcting a luminance of the image on a basis of a periodicchange in the luminance average value; and acquiring color informationon the image and correcting the color information on a basis of aperiodic change in the color information.

According to the present disclosure, there is provided an imaging deviceincluding: an imaging element that captures an image of a subject; andan image processor, in which the image processor includes a firstcorrection section that calculates a luminance average value of theimage and corrects a luminance of the image on a basis of a periodicchange in the luminance average value, and a second correction sectionthat acquires color information on the image and corrects the colorinformation on a basis of a periodic change in the color information.

Effect of the Invention

As described above, according to the present disclosure, it is possibleto suppress a flicker in imaging at a high-speed frame rate.

It is to be noted that the above-described effects are not necessarilylimitative, and any of the effects set forth in the presentspecification or other effects that can be grasped from the presentspecification may be achieved in addition to or in place of theabove-described effects.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schematic view of an imaging device according to anembodiment of the present disclosure.

FIG. 1B is a schematic view of an example in which processing by a firstflicker correction section and processing by a second flicker correctionsection are performed in parallel.

FIG. 2 is a schematic view of an example of using an imaging element inwhich an analog gain is able to be set for each RGB.

FIG. 3 is a schematic view of an example including, in addition to theconfiguration of FIG. 1A, an object detection section that is able todetect a target object and an object tracking section that keepsgrasping a location of the object in a frame after detecting the object.

FIG. 4 is a schematic view of an example of using an imaging element inwhich an analog gain is able to be set for each RGB, similarly to FIG.2.

FIG. 5 is a schematic view of each processing by an exposure amountcalculation section, a first flicker correction section, a secondflicker correction section, and an exposure amount control section.

FIG. 6 is a flowchart illustrating processing of switching correctionmethods depending on whether or not the imaging element is a globalshutter (GS).

FIG. 7A is a flowchart illustrating a flow of processing in theconfiguration example illustrated in FIG. 3.

FIG. 7B is a flowchart illustrating a flow of processing in theconfiguration example illustrated in FIG. 3.

FIG. 8 is a schematic view of flicker correction using an analog gain bythe first flicker correction section and the second flicker correctionsection in a case where the imaging element is the global shutter (GS).

FIG. 9 is a schematic view of flicker correction using an analog gain bythe first flicker correction section and the second flicker correctionsection in a case where the imaging element is a rolling shutter (RS).

FIG. 10 is a schematic view of an example in which an accumulated valuefor each of R, G, and B in one period of a flicker period is used tocomplement, using complementary algorithm, a value of a portion forwhich the accumulated value is not able to be acquired.

FIG. 11 is a schematic view of a method of predicting a correction valuei, and is a schematic view of a complementing method at a frame rate offps and at a light source frequency of Hz.

FIG. 12 is an explanatory schematic view of a method of tracking anobject.

FIG. 13 is a block diagram depicting an example of schematicconfiguration of a vehicle control system.

FIG. 14 is a diagram of assistance in explaining an example ofinstallation positions of an outside-vehicle information detectingsection and an imaging section.

MODES FOR CARRYING OUT THE INVENTION

Hereinafter, description is given in detail of preferred embodiments ofthe present disclosure with reference to the accompanying drawings. Itis to be noted that, in the present specification and drawings, repeateddescription is omitted for components substantially having the samefunctional configuration by assigning the same reference numerals.

It is to be noted that description is given in the following order.

1. Configuration Example of Imaging Device

2. Details of Flicker Correction

3. Regarding Variation of Present Embodiment

4. Regarding Detection and Tracking of Object

5. Application Example

1. Configuration Example of Imaging Device

First, description is given of a configuration of an imaging device 1000according to an embodiment of the present disclosure, with reference toFIG. 1A. As illustrated in FIG. 1A, the imaging device 1000 includes animaging element 100, an A/D conversion section 110, an exposure amountcalculation section 120, a flicker detection section 130, a firstflicker correction section 140, a second flicker correction section 150,a signal processing section 160, and an exposure control section 170.

The imaging element 100 captures an image of a subject at a high-speedframe rate of about 1000 [fps], for example. It is to be noted that, inthe present embodiment, the high-speed frame rate refers to, forexample, a frame rate having a frequency twice or more the frequency ofa flicker phenomenon described later.

The imaging element 100 has a sensor surface on which a plurality ofpixels are arranged in matrix, and outputs a captured image constructedby pixel values corresponding to light receiving amounts of pixelsarranged in an imaging effective region to be utilized in imaging of animage.

Further, in addition to a pixel used for normal imaging, a pixelspecialized in detecting brightness of an imaging environment (alsoreferred to as an OPD (Optical Photo Detector) pixel) may be disposed onthe sensor surface of the imaging element 100. The OPD pixel makes itpossible to detect an OPD value indicating the brightness of the imagingenvironment.

Here, for example, in the imaging element 100, red (R), green (G), andblue (B) color filters are arranged in effective pixels in accordancewith Bayer arrangement. The OPD pixel is colorless in order to detectbrightness. Accordingly, linear interpolation may be performed on pixelvalues of a plurality of effective pixels having the same colors ascolors originally arranged in an arrangement location of the OPD pixeland neighboring the OPD pixel by performing preprocessing on a pixelvalue of each pixel obtained by the imaging element 100, thusdetermining a pixel value of a location corresponding to the arrangementposition of the OPD pixel.

Image data (pixel value) captured by the imaging element 100 isconverted to a digital signal by the A/D conversion section 110, andinputted to the flicker detection section 130. The flicker detectionsection 130 performs flicker detection. This allows for estimation of afrequency of a flicker light source.

Description is given of estimation of flicker period information by theflicker detection section 130. For example, in the present embodiment,an average value of a luminance of an image is stored for each of aplurality of imaging frames, and discrete Fourier transform (DiscreteFourier Transform: DFT) is used for the stored data to thereby estimateinformation on a flicker phenomenon of a frequency, a phase, and thelike.

Particularly, in a case where an object is irradiated with a flickerlight source (e.g., a fluorescent lamp, etc.) that causes a flickerphenomenon, a luminance of the object in the plurality of imaging framesshould vary in accordance with a period of the flicker phenomenon.Therefore, in the present embodiment, an average value of luminances ofa plurality of pixels corresponding to an image of the object areacquired for respective imaging frames acquired at a predetermined framerate. Then, application of the DFT (discrete Fourier transform) allowsfor estimation of a temporal change in a luminance, i.e., a frequencycomponent (frequency, variation in lighting time, lighting interval,etc.) in the temporal change of the luminance.

The imaging frame to be used in estimating period information isacquired at a high-speed frame rate as compared with the period of theflicker phenomenon, thus making it possible to estimate the periodinformation on the flicker phenomenon by application of the DFT.

It is to be noted that, in the present embodiment, the estimation of theperiod information is not limited to the example described above, andanother method may be used. For example, a plurality of imaging framesin a predetermined period of time may be analyzed, and the number offrames from a bright state of the object through a dark state untilreaching a bright state again may be counted, to thereby estimate theperiod information on the flicker phenomenon.

In addition, image data converted to a digital signal by the A/Dconversion section 110 is sequentially inputted to the exposure amountcalculation section 120, the first flicker correction section 140, andthe second flicker correction section 150. FIG. 5 is a schematic view ofeach processing by the exposure amount calculation section 120, thefirst flicker correction section 140, the second flicker correctionsection 150, and the exposure amount control section 170.

The exposure amount calculation section 120 calculates an exposureamount from a detection value and a photometry mode. First, in adetection value acquiring step (step S10), when a screen is divided inton frames, the accumulated value of luminance values and the number ofpixels in each of the frames are acquired. In a photometry modeacquiring step (step S12), weights of the respective frames areacquired. Using these values, an evaluation value is determined from thefollowing expression. It is to be noted that the method of determiningthe evaluation value is not limited to this method; the evaluation valuemay be determined by another calculation method. When the evaluationvalue is determined, determination is made, on a basis of a gap betweenthe evaluation value and a target value, of an exposure amount at whichthe evaluation value converges to the target value.

${{Evaluation}\mspace{14mu}{Value}} = \frac{\Sigma^{Nframe}\left( {n\mspace{14mu}{frame}\mspace{14mu}{weight} \times n\mspace{14mu}{frame}\mspace{14mu}{OPD}\mspace{14mu}{value}} \right)}{\Sigma^{Nframe}\left( {n\mspace{14mu}{frame}\mspace{14mu}{weight} \times n\mspace{14mu}{frame}\mspace{14mu}{pixel}\mspace{14mu}{number}} \right)}$2. Details of Flicker Correction

The first flicker correction section 140 corrects a flicker amount forthe entire screen with respect to the exposure amount calculated in theexposure amount calculation section 120, using the estimated flickerfrequency and luminance value information on the flicker period.Specifically, the first flicker correction section 140 corrects anexposure amount to have a correction amount that does not cause ablocked-up shadow or a blown-out highlight on a picture, and notifiesthe exposure amount control section 170 of the exposure amount, tothereby perform flicker correction using an analog gain.

On the basis of image data captured by the imaging element 100, theexposure amount control section 170 performs an exposure control of theimaging element 100 on the basis of the correction amount calculated infirst flicker correction section 10 for the exposure amount calculatedby the exposure amount calculation section 120 to have an appropriateexposure amount.

FIG. 5 illustrates each processing by the first flicker correctionsection 140, the exposure amount control section 170, and the secondflicker correction section 150, subsequent to the processing by theexposure amount calculation section 120. The first flicker correctionsection 140 calculates a corrected exposure amount in consideration of aflicker correction amount for the exposure amount determined in theexposure amount calculation section 120 (step S16). This correctedexposure amount is used to calculate shutter and gain values in theexposure control section 170, and the values are reflected in theimaging element 100 (step S18). This causes the analog value acquired inthe imaging element 100 to be corrected, thus making it possible toperform correction using an analog gain. This makes it possible tosuppress a blocked-up shadow or a blown-out highlight of an image,unlike correction using a digital gain.

In addition, the second flicker correction section 150 corrects adifference in wavelength characteristics for respective flicker lightsources (characteristics for respective R/G/B) by applying digital gainmodulations that differ for respective RGB s. The second flickercorrection section 150 corrects a difference that has already beencorrected by the first flicker correction section 140. Describing a flowof processing on the basis of FIG. 5, the second flicker correctionsection 150 calculates an exposure amount for flicker correction (stepS20). Then, a correction amount obtained by subtracting a correctionamount in the first flicker correction section 140 is applied (stepS22). An image corrected by the second flicker correction section 150 issubjected to desired signal processing in the signal processing section160, and is outputted as an image signal.

In the processing by the second flicker correction section 150, thecorrection method differs depending on whether or not the imagingelement 100 is a sensor of a global shutter (GS). FIG. 6 is a flowchartillustrating processing of switching correction methods depending onwhether or not the imaging element 100 is the global shutter (GS).

First, in step S30, it is determined whether or not the imaging element100 is a sensor of the global shutter (GS). In a case where the imagingelement 100 is the sensor of the global shutter (GS), the flow proceedsto step S32, where the correction is performed for the entire screen.Meanwhile, the imaging element 100 is not the sensor of the globalshutter (GS) but a sensor of a rolling shutter (RS), the flow proceedsto step S34, where the correction is performed for each line. In stepS36, signal processing is performed by the signal processing section160.

FIG. 8 is a schematic view of flicker correction using a digital gain,performed by the first flicker correction section 140 and the secondflicker correction section 150 in a case where the imaging element 100is the global shutter (GS). In the global shutter (GS), pixel values areread simultaneously throughout all pixels. FIG. 8 schematicallyillustrates, on leftmost side, images of ten frames arranged in avertical direction in a chronological order. In addition, a flickerwaveform and a flicker waveform after correction are illustrated todescribe the processing performed in the first flicker correctionsection 140. In addition, in order to describe the processing performedin the second flicker correction section 150, respective signals of R,G, and B before the correction and respective signals of R, G, and Bafter the correction are illustrated.

In the first flicker correction section 140, an average value d of aflicker waveform b of the respective frames for one period of a flickerperiod is set as a target value of the correction. Here, the flickerwaveform b illustrates a state in which a pixel value of the OPD pixelchanges due to the flicker phenomenon. In a case of the global shutter(GS), the pixel value of the OPD pixel changes for the respectiveframes.

On the basis of the flicker waveform b one period before the flickerperiod, the target value (average value d) and an average value a of theflicker waveform b one period before the flicker period of each of theframes are used to determine a correction value c, which is a differencebetween the average value a and the average value d, for each of theframes. The result of applying the correction value c to the flickerwaveform b is a flicker waveform e after the correction illustrated inFIG. 8. The exposure amount control section 170 performs an exposurecontrol on the basis of the correction amount c to thereby cause, in theflicker waveform e after the correction, the average value a before thecorrection to be moved onto the flicker waveform e. This makes itpossible to suppress fluctuation of luminance components due to theflicker phenomenon.

Next, description is given of flicker correction by the second flickercorrection section 150. As described above, in the second flickercorrection section 150, a correction method is selected depending onwhether the imaging element 100 is the global shutter (GS) or therolling shutter (RS). In FIG. 8, the imaging element 100 is the globalshutter (GS), and thus the flicker correction is performed for each RGBfor the entire screen.

The processing performed by the second flicker correction section 150illustrated in FIG. 8 exemplifies only processing performed on the firstframe for the convenience of description. A waveform f illustrated inFIG. 8 indicates a relationship between a luminance Y before thecorrection by the second flicker correction section 150 and pixel valuesof R, G, and B. The first flicker correction section 140 performscorrection using the luminance Y, whereby the average value a before thecorrection is corrected to the target value for the luminance Y.

Meanwhile, even in a case where the correction is performed by the firstflicker correction section 140, there are differences in the phase andthe amplitude between a luminance signal Y and R, G, B signals.Specifically, the R signal, the G signal, and the B signal after thecorrection by the first flicker correction section 140 are each asindicated by the waveform f illustrated in a state before the correctionof “Processing Performed by Second Flicker Processing Section” in FIG.8. It is appreciated, in the waveform f, that a gap occurs with respectto the target value in the middle for each of the R signal, the Gsignal, and the B signal.

For this reason, the second flicker correction section 150 calculates acorrection amount from RGB values in the entire screen for each RGB, andapplies the correction amount to the entire screen. The processing inthe second flicker correction section 150 is application of thecorrection method in the first flicker correction section 140 to each ofthe R, G, and B. For example, in a case of correcting the R signal,pixel values of the R signal are accumulated for the entire screen ofone frame, and the accumulated value is averaged for one flicker period,to set the averaged value as the target value. When the correction isperformed, the pixel values of the R signal are accumulated for theentire screen of one frame, and a deviation amount from the target valueis set as a correction value, to perform correction by multiplying eachof the pixel values of the R signal by a digital gain corresponding tothe correction value. A correction is performed similarly also for the Bsignal and the G signal. Application of the digital gain to the entirescreen makes it possible, for example, in a case where a subject movesin a portion of a region in the screen, to suppress an influence of themovement on the correction value, thus making it possible to achievehighly robust flicker correction.

At this time, the correction in the first flicker correction section 140is applied to each of the pixels for each of the R, G, and B, and thus agap from the remaining target value obtained by subtracting the amountcorrected by the first flicker correction section 140 is corrected bythe second flicker correction section 150 for each RGB. The result ofthe correction is as indicated by a waveform g illustrated in“Processing Performed by Second Flicker Processing Section” in FIG. 8.As illustrated in the waveform g, it is appreciated that the R signal,the G signal, and the B signal are corrected to the target value in themiddle.

FIG. 9 is a schematic view of flicker correction using an analog gain bythe first flicker correction section 140 and the second flickercorrection section 150 in a case where the imaging element 100 is therolling shutter (RS). In a case of the rolling shutter (RS), pixelvalues are read for each line. In processing illustrated in FIG. 9, thebehavior of the flicker waveform b in the frame differs from that inFIG. 8, but the content of the processing is the same as that in FIG. 8.

In the case of the rolling shutter (RS), exposure is sequentiallyperformed for each line in a horizontal direction in images ofrespective frames of FIG. 9. For this reason, the flicker waveform b foreach line in the horizontal direction of the respective frames due to aflicker phenomenon follows the flicker phenomenon and changes in acurved shape.

Then, as illustrated in FIG. 9, the correction amount c is determinedfrom the entire screen of each of the frames, and correction isperformed for each line. Similarly to FIG. 8, the average value d forone period of the flicker period is set as the target value of thecorrection, and the difference between the average value a and theaverage value d of the flicker waveform b of the respective frames isset as the correction value c. The result of applying the correctionvalue c is the flicker waveform e after the correction illustrated in“Processing Performed by First Flicker Processing Section” illustratedin FIG. 8. The exposure amount control section 170 performs an exposurecontrol on the basis of the correction amount c, to thereby cause, inthe flicker waveform e after the correction, the average value a beforethe correction to be moved onto the flicker waveform e.

In the case of the rolling shutter (RS), the exposure is performed foreach line to cause the pixel values to be read, and thus the flickerwaveform b is obtained in which a luminance of the flicker phenomenon isreflected for each line of each of the frames. In addition, thecorrection value c is uniformly applied to each line of each of theframes, and thus the flicker waveform e after the correction of each ofthe frames has different values for each line.

The second flicker correction section 150 performs processing similar tothat in FIG. 8. That is, the remainder obtained by subtracting theamount corrected by the first flicker correction section 140 iscorrected by the second flicker correction section 150 for each of theR, G, and B. For example, in a case of correcting the R signal, thepixel values of the R signal are accumulated for the entire screen ofone frame, the accumulated value is averaged for one flicker period, toset the averaged value as the target value. Then, the accumulated valueof the pixel values of the R signal of a frame to be corrected isdetermined, and a deviation amount between the target value and theaccumulated value of the R signal of the frame to be corrected is set asa digital gain, to correct each of the pixel values of the R signal. Theresult of the correction is as indicated by the waveform g illustratedin “Processing Performed by Second Flicker Processing Section” in FIG.9.

In the example illustrated in FIG. 9, in the middle line of each of theframes, the luminance signal Y, the R signal, the G signal, and the Bsignal each coincide with a central target value. Meanwhile, also in thewaveform g after the correction by the second flicker correction section150, a deviation between each of the R, G, B signals and the targetvalue occurs as being away from the middle line of each of the frames.

Accordingly, as illustrated in FIG. 10, the accumulated value for eachof the R, G, and B for one period of the flicker period may be used toperform prediction and correction by complementing, using complementaryalgorithm, a value of a portion for which the accumulated value is notable to be acquired.

In FIG. 10, the processing by the first flicker correction section 140is similar to that in FIG. 9. The processing of the second flickercorrection section 150 illustrates processing of correction byexemplifying a case of the R signal. In the processing of the secondflicker correction section 150, a correction amount h is a valueobtained from a deviation amount between the accumulated value of the Rsignal of each of the frames and the target value. Meanwhile, acorrection value i is an estimated value determined by estimation fromthe correction value h of each of the frames. In a case of shooting at ahigh-speed frame rate equal to or more than twice the flicker frequency(typically 100 Hz or 120 Hz), satisfying a sampling theorem makes itpossible to predict the correction value i from the correction value h.

The second flicker correction section 150 uses the correction value hand the correction value i to further correct pixel values of the Rsignal, the G signal, and the B signal that have already been correctedin the first flicker correction section 140.

As described above, in the example illustrated in FIG. 10, a correctionamount for each line is estimated for each RGB from the RGB values ofthe entire surface to perform correction for each line.

FIG. 11 is a schematic view of a method of predicting the correctionvalue i, and illustrates, as an example, an interpolation method in acase of a frame rate of 1000 fps and a light source frequency of 100 Hz.In FIG. 11, an average value of pixel values for each of the R, G, and Bin the screen for each of the frames (a screen average value plotted bya square in the drawing) is determined. Then, an estimated valueindicated by a broken line in FIG. 11 is determined on the basis of thescreen average value. Estimation on the basis of the screen averagevalue is able to be performed using, for example, Lagrange interpolationthat is common as interpolation processing. As illustrated in FIG. 11,it is appreciated that the estimated value repeatedly increases anddecreases periodically in response to the flicker phenomenon.

For example, it is assumed that the screen average value of therespective signals of RGB in the screen up to the tenth frameillustrated in FIG. 11 has already been determined. Assuming that theperiod of the flicker phenomenon corresponds to approximately N frames,the screen average value of the respective signals of the RGB at theeleventh frame and the twelfth frame is able to be determined fromscreen average values of N+1 frame before, N frame before, N−1 framebefore, and N−2 frame before. In addition, the respective signals of RGBfor each line at the eleventh frame and the twelfth frame are able to beestimated from the screen average values of N+1 frame before, N framebefore, N−1 frame before, and N−2 frame before.

As described above, the estimation of a flicker waveform of each lineusing the screen average value of one period before enables a resultthereof to be applied to the correction amount of a current frame. It isto be noted that, in a case where it is not possible to use the screenaverage value of one period before due to a relationship between theframe rate and the light source frequency, an average OPD value of a Mperiod before (M denotes an arbitrary integer) is used.

It is to be noted that, in the case of the rolling shutter (RS), it isnot necessary to perform line-by-line correction. Even in the case ofthe rolling shutter (RS), correction may be performed on ascreen-by-screen basis by determining a correction amount for the entirescreen without determining a correction amount for each line asillustrated in FIG. 9.

It is to be noted that, the description has been given hereinabove ofthe case of the Bayer arrangement; however, even in a case of RCCB orRCCG other than the Bayer arrangement, processing is performedseparately for a luminance and a color similarly to the case of theBayer arrangement, thereby making it possible to perform the flickercorrection.

3. Regarding Variation of Present Embodiment

FIG. 1B is a schematic view of an example in which the processingperformed by the first flicker correction section 140 and the processingperformed by the second flicker correction section 150 are performed inparallel. As illustrated in FIG. 1B, the processing by the first flickercorrection section 140 and the processing by the second flickercorrection section 150 are able to be performed in parallel.

In addition, FIG. 2 is a schematic view of an example of using theimaging element 100 in which an analog gain is able to be set for eachRGB. In a case of using the imaging element 100 in which the analog gainis able to be set for each RGB, it is possible to use a flickercorrection section 180 provided with both of a function of the firstflicker correction section 140 and a function of the second flickercorrection section 150, thus making it unnecessary to use the firstflicker correction section 140 and the second flicker correction section150 separately. In this case, an exposure amount is controlled by theexposure control section 170 for each RGB on the basis of the analoggain for each RGB.

FIG. 3 is a schematic view of an example including, in addition to theconfiguration of FIG. 1A, an object detection section 190 that is ableto detect a target object and an object tracking section 200 that keepsgrasping a location of the object in a frame after detecting the object.In addition, in the example illustrated in FIG. 3, a third flickercorrection section 210 is provided at a subsequent stage of the objecttracking section 200. A function of the third flicker correction section210 is similar to the function of the second flicker correction section150, but differs from the second flicker correction section 150 in thatthe third flicker correction section 210 calculates a correction valuein a predetermined region including a tracked object. In the exampleillustrated in FIG. 3, it is possible for the third flicker correctionsection 210 to calculate a correction amount for each RGB from thevicinity of a focused object and to apply the correction to the entirescreen. The calculation of the correction amount for each RGB from thevicinity of the object makes it possible to perform, with high accuracy,the flicker correction in an object to be focused.

FIG. 4 is a schematic view of an example of using the imaging element100 in which the analog gain is able to be set for each RGB similarly toFIG. 2. In the example illustrated in FIG. 4, the first flickercorrection section 140 and the second flicker correction section 150 arenot provided, and, similarly to FIG. 2, the flicker correction section180 is provided that has both of the function of the first flickercorrection section 140 and the function of the second flicker correctionsection 150.

Further, in the example illustrated in FIG. 4, similarly to FIG. 3, theobject detection section 190 and the object tracking section 200 areprovided, and the third flicker correction section 210 is provided at asubsequent stage of the object tracking section 200. Accordingly, it ispossible to calculate a correction amount for each RGB from the vicinityof a focused object and to apply the correction amount to the entirescreen.

FIG. 7A is a flow chart illustrating a flow of processing in theconfiguration example illustrated in FIG. 3. Hereinafter, description isgiven of the flow of the processing in the configuration exampleillustrated in FIG. 3 on the basis of FIG. 7A. First, in step S40, it isdetermined whether or not the imaging element 100 is the global shutter(GS), and the flow proceeds to step S42 in the case of the globalshutter (GS). In step S42, it is determined whether or not an object hasbeen detected by the object detection section 190.

In a case where the object is detected in step S42, the flow proceeds tostep S44. In step S44, the object is tracked by the object trackingsection 200. In the next step S46, the third flicker correction section210 calculates a correction value for each RGB on the basis of an imagein a predetermined region including the object in the screen.

Meanwhile, in a case where no object is detected in step S42, the flowproceeds to step S48. In step S48, the second flicker correction section140 calculates a correction amount for each RGB from the entire screen.

After steps S46 and S48, the flow proceeds to step S50. In step S50, theflicker correction is performed for the entire screen on the basis ofthe correction values determined in steps S46 and S48. In the next stepS55, the signal processing section performs signal processing. Asdescribed above, in the configuration example illustrated in FIG. 3, ina case where the object detection section 190 detects no object, flickercorrection by the third flicker correction section 210 is not performed,while flicker correction by the second flicker correction section 140 isperformed. In a case where an object is detected from an image after theflicker correction by the second flicker correction section 140, flickercorrection in the vicinity of a region of the object is performed by thethird flicker correction section 210. The detection of an object fromthe image after the flicker correction by the second flicker correctionsection 140 makes it possible to surely detect an object. In addition,in a case where an object is detected from the image after the flickercorrection by the second flicker correction section 140, the flickercorrection in the vicinity of the region of the object performed by thethird flicker correction section 210 makes it possible to perform, withhigh accuracy, the flicker correction in an object to be focused. Whenthe flicker correction by the third flicker correction section 210 isstarted, the flicker correction by the second flicker correction section140 is stopped.

In addition, in a case of not being the global shutter (GS) in step S40,i.e., in the case of the rolling shutter (RS), the flow proceeds to stepS52. In step S52, the second flicker correction section 140 calculates acorrection value for each RGB from the entire screen; in the next stepS54, correction is performed for each line as illustrated in FIG. 10.After steps S50 and S54, the flow proceeds to step S55, where the signalprocessing section 160 performs signal processing.

The processing of the FIG. 7A illustrates the example in which targetdetection is not performed in the case of the rolling shutter (RS);however, as illustrated in the FIG. 7B, object detection may beperformed in the case of the rolling shutter (RS). In the processing ofFIG. 7B, each processing of steps S40, S42, S44, S46, S48, S50, and S55is performed similarly to FIG. 7A. In a case of being determined, instep S40, to be the rolling shutter (RS), the flow proceeds to step S60.In step S60, it is determined whether or not an object has been detectedby the object detection section 190.

In a case where the object has been detected in step S60, the flowproceeds to step S62. In step S62, the object is tracked by the objecttracking section 200. In the next step 64, the third flicker correctionsection 210 calculates a correction value for each RGB on the basis ofan image in a predetermined region including the object in the screen.In the next step S68, the flicker correction is performed for the entirescreen.

Meanwhile, in a case where no object is detected in step S60, the flowproceeds to step S66. In step S66, the second flicker correction section150 calculates a correction amount for each RGB from the entire screen.In the next step S70, the flicker correction is performed for each lineas illustrated in FIG. 10.

After steps S68 and S70, the flow proceeds to step S55. In step S55, theflicker correction is performed for the entire screen on the basis ofthe correction values determined in steps S68 and S70. As describedabove, according to the processing in FIG. 7B, in a case where theimaging element 100 is the rolling shutter (RS), when an object istracked, the flicker correction is performed for each RGB on the basisof an image in a predetermined region including the object. In addition,in the case where the imaging element 100 is the rolling shutter (RS),when an object is not tracked, the correction amount is calculated foreach RGB from the entire screen, and the flicker correction is performedfor each line.

4. Regarding Detection and Tracking of Object

FIG. 12 is a schematic view of an example of detection of an object bythe object detection section 190. The example illustrated in FIG. 12exemplifies a method of detecting an object to be tracked from aplurality of imaging frames captured at a high-speed frame rate. FIG. 12illustrates a human hand as an example of an object. It is assumed thatthe hand is moving slightly in a narrow region for typing on a keyboard.It is to be noted that, in the present embodiment, the object is notlimited to such a human hand.

First, image processing is performed on a captured color imaging frameto generate a grayscaled imaging frame (original images 200 and 202) asillustrated in left side of FIG. 12. Particularly, for example, in acase of generating a grayscaled imaging frame for red (R), pixel valuesof the red (R) of respective pixels in the color imaging frame areextracted as color information. In this case, in the grayscaled imagingframe, a pixel having a high R pixel value becomes white; in a case of alow R pixel value, a pixel becomes black. With respect to the originalimage 200, the position of the hand in the original image 202 of thenext frame has moved,

Then, the R pixel value of each of the pixels of the grayscaled imagingframe is compared with a predetermined threshold value, and, forexample, a pixel value of a pixel having a predetermined threshold valueor more is converted to one, and a pixel value of a pixel less than thepredetermined threshold value is converted to 0. In this manner, it ispossible to generate images 210 and 212 of binarized imaging frames asillustrated in the middle of FIG. 12. In the two binarized images 210and 212 illustrated in the middle of FIG. 12, the pixel value of thehand is converted to one, while the pixel value of the periphery of thehand is converted to zero.

It is to be noted that, in a case of generating a grayscaled imageframe, a pixel value of any of red, green, and blue (RGB) may beextracted, or an average value of three pixel values may be used. Inaddition, after giving weights to the respective pixel values, a valueobtained by accumulating the respective pixel values may be used. Inaddition, when generating the grayscaled imaging frame, it is preferableto select a suitable color and a suitable method depending on featuresof an object to be detected.

Then, the images 210 and 212 of the two binarized imaging framesillustrated in the middle upper tiers of FIG. 12 are compared, and animage 214 is extracted that is obtained by taking a difference betweenthe imaging frames, to thereby extract a specific region (object) inwhich a movement has occurred in the image 214. In the image 214, aregion in which the hand is moved has a pixel value difference of one,and thus is illustrated in white. Accordingly, in the image 214, theregion illustrated in white is detected as the movement of the object(hand). Then, repeating such processing makes it possible to track theobject.

It is to be noted that, in the present embodiment, the detection of theobject is not limited to the above-described methods; for example,feature points of the object may be stored in advance, and the storedfeature points may be extracted from the imaging frame, to therebyperform detection of the object. For example, in this case, when theobject is a face of a person, detection of the person is performed onthe basis of feature points (eyes, nose, mouth) of the face of theperson.

Next, description is given in detail of tracking of an object by theobject tracking section 200. As illustrated in an image 216 on rightside of FIG. 12, the object tracking section 200 performs tracking(tracking) for the detected object (hand).

For example, a self-window method is used to keep tracking an object ina plurality of successive imaging frames captured at a high-speed framerate. The self-window method is one type of algorithm for tracking anobject in imaging frames captured at a high-speed frame rate. In a caseof imaging at a high-speed frame rate, a moving distance (difference) ofthe object in the plurality of imaging frames is small. Accordingly, ina case where an extraction window for extracting the object is set in aregion around the object in a previous imaging frame, the object isincluded in the extraction window even in the next imaging frame.

More specifically, the extraction window is set as a pixel region thatis expanded outward by one pixel with respect to a pixel regionindicating the object in the previous imaging frame. In a case where theobject is not included in the set extraction window in the next imagingframe rate, the frame rate is further sped up to thereby enable theobject to be included in the extraction window. In addition, the movingdistance of the object is small, and thus it is possible to decrease anarea of the extraction window, which is a range in which the object issearched. Accordingly, performing image matching or the like in theextraction window makes it possible to easily detect the object. In theexample of FIG. 12, it is possible to track the object using a skincolor for detection of the hand, on the basis of whether or not pixelvalues of R, G, and B constituting the skin color are present in theextraction window.

It is to be noted that the extraction window has been described as beingset as the pixel region that is expanded outward by one pixel withrespect to the pixel region indicating the object in the previousimaging frame; however, the present embodiment is not limited toexpanding by one pixel. For example, a pixel region expanded by two ormore pixels may be set as the extraction window. The number of pixels tobe expanded may be appropriately selected depending on, for example, theframe rate, the speed of the object, and the like.

In addition, in a case where the object is not included in the setextraction window in the next imaging frame rate, the frame rate isfurther sped up, but such a method is not limitative. For example, thenumber of pixels to be expanded when setting the extraction window maybe adjusted to thereby cause the object to be included in the extractionwindow.

Further, the self-window method involves calculating a logical productbetween the extraction window calculated from the previous imaging frameand a next binarized object image to thereby generate a targetextraction image and to track the object. In addition, according to theself-window method, images of the object between target extractionimages of preceding and succeeding imaging frames are compared, therebymaking it possible to acquire not only position information and area(shape) information on the object but also information such as a movingdirection and a moving speed of the object. As is obvious from the abovedescription, in a case of using the self-window method, it is preferablethat the imaging frame for use in tracking the object be captured at ahigh-speed frame rate.

It is to be noted that, in the present embodiment, the tracking of theobject is not limited to the above-described examples, and anothermethod may be used. For example, in the present embodiment, the trackingof the object may be performed on the basis of feature points common topreceding and succeeding imaging frames.

As described above, according to the present embodiment, causing thefirst flicker correction section 140 to perform the flicker correctionmakes it possible to suppress occurrence of saturation (a blown-outhighlight or a blocked-up shadow) at a time point of analog dataacquired by the imaging element 100, and thus to increase S/N. Inaddition, it is possible to provide a robust flicker correction methodeven for the movement of the imaging element 100 and for a movingsubject.

4. Application Example

The technology according to an embodiment of the present disclosure isapplicable to a variety of products. For example, the technologyaccording to an embodiment of the present disclosure may be implementedas a device to be mounted on a mobile body of any kind, such as anautomobile, an electric vehicle, a hybrid electric vehicle, amotorcycle, a bicycle, any personal mobility device, an airplane, adrone, a vessel, a robot, a construction machine, and an agriculturalmachine (tractor).

FIG. 13 is a block diagram depicting an example of schematicconfiguration of a vehicle control system 7000 as an example of a mobilebody control system to which the technology according to an embodimentof the present disclosure can be applied. The vehicle control system7000 includes a plurality of electronic control units connected to eachother via a communication network 7010. In the example depicted in FIG.13, the vehicle control system 7000 includes a driving system controlunit 7100, a body system control unit 7200, a battery control unit 7300,an outside-vehicle information detecting unit 7400, an in-vehicleinformation detecting unit 7500, and an integrated control unit 7600.The communication network 7010 connecting the plurality of control unitsto each other may, for example, be a vehicle-mounted communicationnetwork compliant with an arbitrary standard such as controller areanetwork (CAN), local interconnect network (LIN), local area network(LAN), FlexRay (registered trademark), or the like.

Each of the control units includes: a microcomputer that performsarithmetic processing according to various kinds of programs; a storagesection that stores the programs executed by the microcomputer,parameters used for various kinds of operations, or the like; and adriving circuit that drives various kinds of control target devices.Each of the control units further includes: a network interface (I/F)for performing communication with other control units via thecommunication network 7010; and a communication I/F for performingcommunication with a device, a sensor, or the like within and withoutthe vehicle by wire communication or radio communication. A functionalconfiguration of the integrated control unit 7600 illustrated in FIG. 13includes a microcomputer 7610, a general-purpose communication I/F 7620,a dedicated communication I/F 7630, a positioning section 7640, a beaconreceiving section 7650, an in-vehicle device I/F 7660, a sound/imageoutput section 7670, a vehicle-mounted network I/F 7680, and a storagesection 7690. The other control units similarly include a microcomputer,a communication I/F, a storage section, and the like.

The driving system control unit 7100 controls the operation of devicesrelated to the driving system of the vehicle in accordance with variouskinds of programs. For example, the driving system control unit 7100functions as a control device for a driving force generating device forgenerating the driving force of the vehicle, such as an internalcombustion engine, a driving motor, or the like, a driving forcetransmitting mechanism for transmitting the driving force to wheels, asteering mechanism for adjusting the steering angle of the vehicle, abraking device for generating the braking force of the vehicle, and thelike. The driving system control unit 7100 may have a function as acontrol device of an antilock brake system (ABS), electronic stabilitycontrol (ESC), or the like.

The driving system control unit 7100 is connected with a vehicle statedetecting section 7110. The vehicle state detecting section 7110, forexample, includes at least one of a gyro sensor that detects the angularvelocity of axial rotational movement of a vehicle body, an accelerationsensor that detects the acceleration of the vehicle, and sensors fordetecting an amount of operation of an accelerator pedal, an amount ofoperation of a brake pedal, the steering angle of a steering wheel, anengine speed or the rotational speed of wheels, and the like. Thedriving system control unit 7100 performs arithmetic processing using asignal input from the vehicle state detecting section 7110, and controlsthe internal combustion engine, the driving motor, an electric powersteering device, the brake device, and the like.

The body system control unit 7200 controls the operation of variouskinds of devices provided to the vehicle body in accordance with variouskinds of programs. For example, the body system control unit 7200functions as a control device for a keyless entry system, a smart keysystem, a power window device, or various kinds of lamps such as aheadlamp, a backup lamp, a brake lamp, a turn signal, a fog lamp, or thelike. In this case, radio waves transmitted from a mobile device as analternative to a key or signals of various kinds of switches can beinput to the body system control unit 7200. The body system control unit7200 receives these input radio waves or signals, and controls a doorlock device, the power window device, the lamps, or the like of thevehicle.

The battery control unit 7300 controls a secondary battery 7310, whichis a power supply source for the driving motor, in accordance withvarious kinds of programs. For example, the battery control unit 7300 issupplied with information about a battery temperature, a battery outputvoltage, an amount of charge remaining in the battery, or the like froma battery device including the secondary battery 7310. The batterycontrol unit 7300 performs arithmetic processing using these signals,and performs control for regulating the temperature of the secondarybattery 7310 or controls a cooling device provided to the battery deviceor the like.

The outside-vehicle information detecting unit 7400 detects informationabout the outside of the vehicle including the vehicle control system7000. For example, the outside-vehicle information detecting unit 7400is connected with at least one of an imaging section 7410 and anoutside-vehicle information detecting section 7420. The imaging section7410 includes at least one of a time-of-flight (ToF) camera, a stereocamera, a monocular camera, an infrared camera, and other cameras. Theoutside-vehicle information detecting section 7420, for example,includes at least one of an environmental sensor for detecting currentatmospheric conditions or weather conditions and a peripheralinformation detecting sensor for detecting another vehicle, an obstacle,a pedestrian, or the like on the periphery of the vehicle including thevehicle control system 7000.

The environmental sensor, for example, may be at least one of a raindrop sensor detecting rain, a fog sensor detecting a fog, a sunshinesensor detecting a degree of sunshine, and a snow sensor detecting asnowfall. The peripheral information detecting sensor may be at leastone of an ultrasonic sensor, a radar device, and a LIDAR device (Lightdetection and Ranging device, or Laser imaging detection and rangingdevice). Each of the imaging section 7410 and the outside-vehicleinformation detecting section 7420 may be provided as an independentsensor or device, or may be provided as a device in which a plurality ofsensors or devices are integrated.

FIG. 14 depicts an example of installation positions of the imagingsection 7410 and the outside-vehicle information detecting section 7420.Imaging sections 7910, 7912, 7914, 7916, and 7918 are, for example,disposed at at least one of positions on a front nose, sideview mirrors,a rear bumper, and a back door of the vehicle 7900 and a position on anupper portion of a windshield within the interior of the vehicle. Theimaging section 7910 provided to the front nose and the imaging section7918 provided to the upper portion of the windshield within the interiorof the vehicle obtain mainly an image of the front of the vehicle 7900.The imaging sections 7912 and 7914 provided to the sideview mirrorsobtain mainly an image of the sides of the vehicle 7900. The imagingsection 7916 provided to the rear bumper or the back door obtains mainlyan image of the rear of the vehicle 7900. The imaging section 7918provided to the upper portion of the windshield within the interior ofthe vehicle is used mainly to detect a preceding vehicle, a pedestrian,an obstacle, a signal, a traffic sign, a lane, or the like.

Incidentally, FIG. 14 depicts an example of photographing ranges of therespective imaging sections 7910, 7912, 7914, and 7916. An imaging rangea represents the imaging range of the imaging section 7910 provided tothe front nose. Imaging ranges b and c respectively represent theimaging ranges of the imaging sections 7912 and 7914 provided to thesideview mirrors. An imaging range d represents the imaging range of theimaging section 7916 provided to the rear bumper or the back door. Abird's-eye image of the vehicle 7900 as viewed from above can beobtained by superimposing image data imaged by the imaging sections7910, 7912, 7914, and 7916, for example.

Outside-vehicle information detecting sections 7920, 7922, 7924, 7926,7928, and 7930 provided to the front, rear, sides, and corners of thevehicle 7900 and the upper portion of the windshield within the interiorof the vehicle may be, for example, an ultrasonic sensor or a radardevice. The outside-vehicle information detecting sections 7920, 7926,and 7930 provided to the front nose of the vehicle 7900, the rearbumper, the back door of the vehicle 7900, and the upper portion of thewindshield within the interior of the vehicle may be a LIDAR device, forexample. These outside-vehicle information detecting sections 7920 to7930 are used mainly to detect a preceding vehicle, a pedestrian, anobstacle, or the like.

Returning to FIG. 13, the description will be continued. Theoutside-vehicle information detecting unit 7400 makes the imagingsection 7410 image an image of the outside of the vehicle, and receivesimaged image data. In addition, the outside-vehicle informationdetecting unit 7400 receives detection information from theoutside-vehicle information detecting section 7420 connected to theoutside-vehicle information detecting unit 7400. In a case where theoutside-vehicle information detecting section 7420 is an ultrasonicsensor, a radar device, or a LIDAR device, the outside-vehicleinformation detecting unit 7400 transmits an ultrasonic wave, anelectromagnetic wave, or the like, and receives information of areceived reflected wave. On the basis of the received information, theoutside-vehicle information detecting unit 7400 may perform processingof detecting an object such as a human, a vehicle, an obstacle, a sign,a character on a road surface, or the like, or processing of detecting adistance thereto. The outside-vehicle information detecting unit 7400may perform environment recognition processing of recognizing arainfall, a fog, road surface conditions, or the like on the basis ofthe received information. The outside-vehicle information detecting unit7400 may calculate a distance to an object outside the vehicle on thebasis of the received information.

In addition, on the basis of the received image data, theoutside-vehicle information detecting unit 7400 may perform imagerecognition processing of recognizing a human, a vehicle, an obstacle, asign, a character on a road surface, or the like, or processing ofdetecting a distance thereto. The outside-vehicle information detectingunit 7400 may subject the received image data to processing such asdistortion correction, alignment, or the like, and combine the imagedata imaged by a plurality of different imaging sections 7410 togenerate a bird's-eye image or a panoramic image. The outside-vehicleinformation detecting unit 7400 may perform viewpoint conversionprocessing using the image data imaged by the imaging section 7410including the different imaging parts.

The in-vehicle information detecting unit 7500 detects information aboutthe inside of the vehicle. The in-vehicle information detecting unit7500 is, for example, connected with a driver state detecting section7510 that detects the state of a driver. The driver state detectingsection 7510 may include a camera that images the driver, a biosensorthat detects biological information of the driver, a microphone thatcollects sound within the interior of the vehicle, or the like. Thebiosensor is, for example, disposed in a seat surface, the steeringwheel, or the like, and detects biological information of an occupantsitting in a seat or the driver holding the steering wheel. On the basisof detection information input from the driver state detecting section7510, the in-vehicle information detecting unit 7500 may calculate adegree of fatigue of the driver or a degree of concentration of thedriver, or may determine whether the driver is dozing. The in-vehicleinformation detecting unit 7500 may subject an audio signal obtained bythe collection of the sound to processing such as noise cancelingprocessing or the like.

The integrated control unit 7600 controls general operation within thevehicle control system 7000 in accordance with various kinds ofprograms. The integrated control unit 7600 is connected with an inputsection 7800. The input section 7800 is implemented by a device capableof input operation by an occupant, such, for example, as a touch panel,a button, a microphone, a switch, a lever, or the like. The integratedcontrol unit 7600 may be supplied with data obtained by voicerecognition of voice input through the microphone. The input section7800 may, for example, be a remote control device using infrared rays orother radio waves, or an external connecting device such as a mobiletelephone, a personal digital assistant (PDA), or the like that supportsoperation of the vehicle control system 7000. The input section 7800 maybe, for example, a camera. In that case, an occupant can inputinformation by gesture. Alternatively, data may be input which isobtained by detecting the movement of a wearable device that an occupantwears. Further, the input section 7800 may, for example, include aninput control circuit or the like that generates an input signal on thebasis of information input by an occupant or the like using theabove-described input section 7800, and which outputs the generatedinput signal to the integrated control unit 7600. An occupant or thelike inputs various kinds of data or gives an instruction for processingoperation to the vehicle control system 7000 by operating the inputsection 7800.

The storage section 7690 may include a read only memory (ROM) thatstores various kinds of programs executed by the microcomputer and arandom access memory (RAM) that stores various kinds of parameters,operation results, sensor values, or the like. In addition, the storagesection 7690 may be implemented by a magnetic storage device such as ahard disc drive (HDD) or the like, a semiconductor storage device, anoptical storage device, a magneto-optical storage device, or the like.

The general-purpose communication I/F 7620 is a communication I/F usedwidely, which communication I/F mediates communication with variousapparatuses present in an external environment 7750. The general-purposecommunication I/F 7620 may implement a cellular communication protocolsuch as global system for mobile communications (GSM (registeredtrademark)), worldwide interoperability for microwave access (WiMAX(registered trademark)), long term evolution (LTE (registeredtrademark)), LTE-advanced (LTE-A), or the like, or another wirelesscommunication protocol such as wireless LAN (referred to also aswireless fidelity (Wi-Fi (registered trademark)), Bluetooth (registeredtrademark), or the like. The general-purpose communication I/F 7620 may,for example, connect to an apparatus (for example, an application serveror a control server) present on an external network (for example, theInternet, a cloud network, or a company-specific network) via a basestation or an access point. In addition, the general-purposecommunication I/F 7620 may connect to a terminal present in the vicinityof the vehicle (which terminal is, for example, a terminal of thedriver, a pedestrian, or a store, or a machine type communication (MTC)terminal) using a peer to peer (P2P) technology, for example.

The dedicated communication I/F 7630 is a communication I/F thatsupports a communication protocol developed for use in vehicles. Thededicated communication I/F 7630 may implement a standard protocol such,for example, as wireless access in vehicle environment (WAVE), which isa combination of institute of electrical and electronic engineers (IEEE)802.11p as a lower layer and IEEE 1609 as a higher layer, dedicatedshort range communications (DSRC), or a cellular communication protocol.The dedicated communication I/F 7630 typically carries out V2Xcommunication as a concept including one or more of communicationbetween a vehicle and a vehicle (Vehicle to Vehicle), communicationbetween a road and a vehicle (Vehicle to Infrastructure), communicationbetween a vehicle and a home (Vehicle to Home), and communicationbetween a pedestrian and a vehicle (Vehicle to Pedestrian).

The positioning section 7640, for example, performs positioning byreceiving a global navigation satellite system (GNSS) signal from a GNSSsatellite (for example, a GPS signal from a global positioning system(GPS) satellite), and generates positional information including thelatitude, longitude, and altitude of the vehicle. Incidentally, thepositioning section 7640 may identify a current position by exchangingsignals with a wireless access point, or may obtain the positionalinformation from a terminal such as a mobile telephone, a personalhandyphone system (PHS), or a smart phone that has a positioningfunction.

The beacon receiving section 7650, for example, receives a radio wave oran electromagnetic wave transmitted from a radio station installed on aroad or the like, and thereby obtains information about the currentposition, congestion, a closed road, a necessary time, or the like.Incidentally, the function of the beacon receiving section 7650 may beincluded in the dedicated communication I/F 7630 described above.

The in-vehicle device I/F 7660 is a communication interface thatmediates connection between the microcomputer 7610 and variousin-vehicle devices 7760 present within the vehicle. The in-vehicledevice I/F 7660 may establish wireless connection using a wirelesscommunication protocol such as wireless LAN, Bluetooth (registeredtrademark), near field communication (NFC), or wireless universal serialbus (WUSB). In addition, the in-vehicle device I/F 7660 may establishwired connection by universal serial bus (USB), high-definitionmultimedia interface (HDMI (registered trademark)), mobilehigh-definition link (MHL), or the like via a connection terminal (and acable if necessary) not depicted in the figures. The in-vehicle devices7760 may, for example, include at least one of a mobile device and awearable device possessed by an occupant and an information devicecarried into or attached to the vehicle. The in-vehicle devices 7760 mayalso include a navigation device that searches for a path to anarbitrary destination. The in-vehicle device I/F 7660 exchanges controlsignals or data signals with these in-vehicle devices 7760.

The vehicle-mounted network I/F 7680 is an interface that mediatescommunication between the microcomputer 7610 and the communicationnetwork 7010. The vehicle-mounted network I/F 7680 transmits andreceives signals or the like in conformity with a predetermined protocolsupported by the communication network 7010.

The microcomputer 7610 of the integrated control unit 7600 controls thevehicle control system 7000 in accordance with various kinds of programson the basis of information obtained via at least one of thegeneral-purpose communication I/F 7620, the dedicated communication I/F7630, the positioning section 7640, the beacon receiving section 7650,the in-vehicle device I/F 7660, and the vehicle-mounted network I/F7680. For example, the microcomputer 7610 may calculate a control targetvalue for the driving force generating device, the steering mechanism,or the braking device on the basis of the obtained information about theinside and outside of the vehicle, and output a control command to thedriving system control unit 7100. For example, the microcomputer 7610may perform cooperative control intended to implement functions of anadvanced driver assistance system (ADAS) which functions includecollision avoidance or shock mitigation for the vehicle, followingdriving based on a following distance, vehicle speed maintainingdriving, a warning of collision of the vehicle, a warning of deviationof the vehicle from a lane, or the like. In addition, the microcomputer7610 may perform cooperative control intended for automatic driving,which makes the vehicle to travel autonomously without depending on theoperation of the driver, or the like, by controlling the driving forcegenerating device, the steering mechanism, the braking device, or thelike on the basis of the obtained information about the surroundings ofthe vehicle.

The microcomputer 7610 may generate three-dimensional distanceinformation between the vehicle and an object such as a surroundingstructure, a person, or the like, and generate local map informationincluding information about the surroundings of the current position ofthe vehicle, on the basis of information obtained via at least one ofthe general-purpose communication I/F 7620, the dedicated communicationI/F 7630, the positioning section 7640, the beacon receiving section7650, the in-vehicle device I/F 7660, and the vehicle-mounted networkI/F 7680. In addition, the microcomputer 7610 may predict danger such ascollision of the vehicle, approaching of a pedestrian or the like, anentry to a closed road, or the like on the basis of the obtainedinformation, and generate a warning signal. The warning signal may, forexample, be a signal for producing a warning sound or lighting a warninglamp.

The sound/image output section 7670 transmits an output signal of atleast one of a sound and an image to an output device capable ofvisually or auditorily notifying information to an occupant of thevehicle or the outside of the vehicle. In the example of FIG. 13, anaudio speaker 7710, a display section 7720, and an instrument panel 7730are illustrated as the output device. The display section 7720 may, forexample, include at least one of an on-board display and a head-updisplay. The display section 7720 may have an augmented reality (AR)display function. The output device may be other than these devices, andmay be another device such as headphones, a wearable device such as aneyeglass type display worn by an occupant or the like, a projector, alamp, or the like. In a case where the output device is a displaydevice, the display device visually displays results obtained by variouskinds of processing performed by the microcomputer 7610 or informationreceived from another control unit in various forms such as text, animage, a table, a graph, or the like. In addition, in a case where theoutput device is an audio output device, the audio output deviceconverts an audio signal constituted of reproduced audio data or sounddata or the like into an analog signal, and auditorily outputs theanalog signal.

Incidentally, at least two control units connected to each other via thecommunication network 7010 in the example depicted in FIG. 13 may beintegrated into one control unit. Alternatively, each individual controlunit may include a plurality of control units. Further, the vehiclecontrol system 7000 may include another control unit not depicted in thefigures. In addition, part or the whole of the functions performed byone of the control units in the above description may be assigned toanother control unit. That is, predetermined arithmetic processing maybe performed by any of the control units as long as information istransmitted and received via the communication network 7010. Similarly,a sensor or a device connected to one of the control units may beconnected to another control unit, and a plurality of control units maymutually transmit and receive detection information via thecommunication network 7010.

In the vehicle control system 7000 described above, the imaging device1000, an information processor 100 according to the present embodimentdescribed with reference to FIGS. 1A and 1B, etc. may be applied to theimaging section 7410 of the application example illustrated in FIG. 13.

Although the description has been given above in detail of preferredembodiments of the present disclosure with reference to the accompanyingdrawings, the technical scope of the present disclosure is not limitedto such examples. It is obvious that a person having ordinary skill inthe art of the present disclosure may find various alterations ormodifications within the scope of the technical idea described in theclaims, and it should be understood that these alterations andmodifications naturally come under the technical scope of the presentdisclosure.

In addition, the effects described herein are merely illustrative orexemplary, and are not limitative. That is, the technique according tothe present disclosure may achieve, in addition to or in place of theabove effects, other effects that are obvious to those skilled in theart from the description of the present specification.

It is to be noted that the technical scope of the present disclosurealso includes the following configurations.

(1)

An image processor including:

a first correction section that calculates a luminance average value ofan image and corrects a luminance of the image on a basis of a periodicchange in the luminance average value; and

a second correction section that acquires color information on the imageand corrects the color information on a basis of a periodic change inthe color information.

(2)

The image processor according to (1), in which the first correctionsection sets, as a target value, a value obtained by averaging therespective luminance average values in a plurality of frames, andperforms correction to cause the luminance of the image to be the targetvalue.

(3)

The image processor according to (2), in which the plurality of framesincludes at least one period of a luminance change due to a flickerphenomenon.

(4)

The image processor according to any one of (1) to (3), including anexposure amount control section that controls an exposure amount of theimage on a basis of a correction value calculated by the firstcorrection section on the basis of the periodic change in the luminanceaverage value.

(5)

The image processor according to any one of (1) to (4), in which thesecond correction section performs correction for the image subjected tothe correction by the first correction section.

(6)

The image processor according to any one of (1) to (5), in which thesecond correction section determines respective accumulated values ofvalues of a plurality of pieces of the color information on the image,sets, as a target value, a value obtained by averaging the accumulatedvalues in a plurality of frames, and corrects each of the plurality ofpieces of the color information to cause each of the accumulated valuesto be the target value.

(7)

The image processor according to (6), in which the plurality of framesincludes at least one period of a change in the color information due tothe flicker phenomenon.

(8)

The image processor according to (6), in which the plurality of piecesof the color information includes the color information of three colorsof red, green and blue.

(9)

The image processor according to any one of (1) to (8), in which thesecond correction section corrects the color information uniformlythroughout the image.

(10)

The image processor according to any one of (1) to (8), in which thesecond correction section performs correction for each line of pixelarrangement of the color information.

(11)

The image processor according to (6), in which

the second correction section performs correction on a basis of theaccumulated values for an optional line in one frame, and

the second correction section performs correction on a basis of anestimated value estimated from the accumulated values for a line otherthan the optional line.

(12)

The image processor according to any one of (1) to (11), including:

an object detection section that detects an object in the image; and

a tracking section that performs tracking of the object,

the second correction section correcting the color information in apredetermined region including the object subjected to the tracking.

(13)

An image processing method including:

calculating a luminance average value of an image and correcting aluminance of the image on a basis of a periodic change in the luminanceaverage value; and

acquiring color information on the image and correcting the colorinformation on a basis of a periodic change in the color information.

(14)

An imaging device including:

an imaging element that captures an image of a subject; and

an image processor,

the image processor including

-   -   a first correction section that calculates a luminance average        value of the image and corrects a luminance of the image on a        basis of a periodic change in the luminance average value, and    -   a second correction section that acquires color information on        the image and corrects the color information on a basis of a        periodic change in the color information.

REFERENCE NUMERALS LIST

-   100 imaging element-   140 first flicker correction section-   150 second flicker correction section-   170 exposure control section-   1000 imaging device

The invention claimed is:
 1. An image processor, comprising: a firstcorrection section configured to: calculate a luminance average value ofan image; and correct a luminance of the image based on a periodicchange in the luminance average value; and a second correction sectionconfigured to: acquire color information of the image; determineaccumulated values for each of a plurality of pieces of the colorinformation of the image; set, as a target value, a value that isobtained based on average of the accumulated values for one of theplurality of pieces of the color information in a plurality of frames ofthe image; and correct each of the plurality of pieces of the colorinformation to cause each of the accumulated values to be the targetvalue.
 2. The image processor according to claim 1, wherein the firstcorrection section is further configured to: set, as the target value, avalue that is obtained based on an average of respective luminanceaverage values in the plurality of frames; and perform correction tocause the luminance of the image to be the target value.
 3. The imageprocessor according to claim 2, wherein the plurality of frames includesat least one period of a luminance change due to a flicker phenomenon.4. The image processor according to claim 1, further comprising anexposure amount control section configured to control an exposure amountof the image based on a correction value calculated by the firstcorrection section, wherein the correction value is calculated based onthe periodic change in the luminance average value.
 5. The imageprocessor according to claim 1, wherein the second correction section isfurther configured to perform correction for the image subjected to thecorrection by the first correction section.
 6. The image processoraccording to claim 1, wherein the plurality of frames includes at leastone period of a change in the color information due to a flickerphenomenon.
 7. The image processor according to claim 1, wherein theplurality of pieces of the color information comprises the colorinformation of at least one of a red color, a green color, or a bluecolor.
 8. The image processor according to claim 1, wherein the secondcorrection section is further configured to correct the colorinformation uniformly throughout the image.
 9. The image processoraccording to claim 1, wherein the second correction section is furtherconfigured to perform correction for each line of pixel arrangement ofthe color information.
 10. The image processor according to claim 1,wherein the second correction section is further configured to: performcorrection based on the accumulated values for an optional line in oneframe; and perform correction based on an estimated value estimated fromthe accumulated values for a line different from the optional line. 11.The image processor according to claim 1, further comprising: an objectdetection section configured to detect an object in the image; and atracking section configured to track the object, wherein the secondcorrection section is further configured to correct the colorinformation in a region that includes the tracked object.
 12. An imageprocessing method, comprising: calculating a luminance average value ofan image; correcting a luminance of the image based on a periodic changein the luminance average value; acquiring color information of theimage; determining accumulated values for each of a plurality of piecesof the color information of the image; setting, as a target value, avalue that is obtained based on average of the accumulated values forone of the plurality of pieces of the color information in a pluralityof frames of the image; and correcting each of the plurality of piecesof the color information to cause each of the accumulated values to bethe target value.
 13. An imaging device, comprising: an imaging elementconfigured to capture an image of a subject; and an image processor,wherein the image processor includes: a first correction sectionconfigured to: calculate a luminance average value of the image; andcorrect a luminance of the image based on a periodic change in theluminance average value, and a second correction section configured to:acquire color information of the image; determine accumulated values foreach of a plurality of pieces of the color information of the image;set, as a target value, a value that is obtained based on average of theaccumulated values for one of the plurality of pieces of the colorinformation in a plurality of frames of the image; and correct each ofthe plurality of pieces of the color information to cause each of theaccumulated values to be the target value.
 14. The image processoraccording to claim 1, further comprising: a third correction sectionconfigured to: calculate a new correction value for each of theplurality of pieces of the color information from a vicinity of anobject in the image; and perform, based on the new correction value,correction of the image subjected to the correction by the firstcorrection section and the second correction section.